Media content streaming using stream message fragments

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for media content streaming can include transacting access information associated with a media stream and transacting one or more fragments associated with the media stream to facilitate a delivery of media content associated with the media stream. Access information can include fragment sequencing information to facilitate individual retrieval of fragments associated with the media stream using a uniform resource identifier via a processing device configured to cache content. A fragment can include one or more stream messages. A stream message can include a message header and a corresponding media data sample. The message header can include a message stream identifier, a message type identifier, a timestamp, and a message length value.

RELATED APPLICATIONS

This Application claims priority under 35 U.S.C. Section 120 as acontinuation of U.S. patent application Ser. No. 12/542,552, filed Aug.17, 2009, and titled Media Content Streaming, the entire disclosure ofwhich is hereby incorporated by reference in its entirety.

BACKGROUND

This specification relates to streaming media content.

A networked server can stream media content to one or more computersconnected to a communication network. Various examples of media contentinclude video, audio, text, and combinations thereof. A computer canrequest and receive a media content. A computer can render media contentto an output device such as a video display, speaker, or a printer.

Processing devices such as a computer or a server can use one or moreprotocols to exchange media content. For example, some processingdevices can use a protocol such as the Real-Time Messaging Protocol(RTMP) of Adobe Systems Incorporated of San Jose, Calif. to send mediacontent over a network such as one based on an Internet Protocol (IP).RTMP can provide multiplexing and packetizing services for ahigher-level multimedia stream protocol. RTMP messages can include atimestamp and payload type identification information. Protocols such asRTMP can use a reliable transport protocol such as Transmission ControlProtocol (TCP) to provide guaranteed timestamp-ordered end-to-enddelivery of messages, across one or more streams.

Some processing devices can use a Web protocol such as a HypertextTransfer Protocol (HTTP) to request and receive information such as adocument or at least a portion of a document. Some HTTP requests canidentify a specific document. Processing devices can transact HTTP dataover TCP/IP.

SUMMARY

This specification describes technologies relating to Web based mediacontent streaming.

In one aspect, methods for media content streaming can includetransacting access information associated with a media stream andtransacting one or more fragments associated with the media stream tofacilitate a delivery of media content associated with the media stream.Access information can include fragment sequencing information tofacilitate individual retrieval of fragments associated with the mediastream using a uniform resource identifier via a processing deviceconfigured to cache content. A fragment can include one or more streammessages. A stream message can include a message header and acorresponding media data sample. The message header can include amessage stream identifier, a message type identifier, a timestamp, and amessage length value. Other implementations can include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices.

These and other implementations can include one or more of the followingfeatures. Transacting one or more of the fragments associated with themedia stream can include sending the one or more of the fragments to theprocessing device. A processing device can be configured to cachefragments associated with the media stream and to deliver cachedfragments to a remote device using a Hypertext Transfer Protocol (HTTP).Access information can include identities of multiple servers, includingthe processing device, configured to cache one or more fragmentsassociated with the media stream. These and other implementations caninclude causing the remote device to request different fragments fromdifferent ones of the servers, and to assemble requested fragments toplay at least a portion of the media stream.

These and other implementations can include producing one or moreadditional fragments based on an incremental media data addition to themedia stream, providing updated access information to a remote device toreflect the one or more additional fragments, and causing the processingdevice to cache the one or more additional fragments. Fragmentsequencing information can include one or more fragment time durationsand a number of fragments associated with each fragment time duration.Fragment sequencing information can be arranged to indicate a fragmentplay order. A message header can be formatted in accordance with aReal-Time Messaging Protocol (RTMP).

These and other implementations can include comprising causing a remotedevice to process media data based on the information contained in atleast one message header. One or more of the fragments can include aRTMP message associated with audio data interleaved with a RTMP messageassociated with video data. Access information can include a segment runtable to identify runs of segments and a fragment run table to identifyruns of fragments. The fragment run tables can include the fragmentsequencing information.

In another aspect, methods for media content streaming can includecausing a server cluster to store media content. The media content caninclude fragments associated with a media stream. A fragment can includestream messages, where separate ones of the stream message can include amessage header and a corresponding media data sample. A message headercan include a message stream identifier, a message type identifier, atimestamp, and a message length value. These methods can includereceiving a request associated with the media stream from a remotedevice using a Web protocol. These methods can include sending accessinformation associated with the media stream to the remote device usinga Web protocol. The access information can include fragment sequencinginformation to facilitate individual fragment retrieval by the remotedevice. These methods can include causing the server cluster to processa fragment request from a remote device that identifies one or more offragments associated with a media stream, and to send the one or moreidentified fragments to the remote device using a Web protocol. Otherimplementations can include corresponding systems, apparatus, andcomputer programs, configured to perform the actions of the methods,encoded on computer storage devices.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Media content streaming throughput can beincreased. HTTP can be used to stream media content, and HTTP networkinfrastructure and HTTP client software can be leveraged. Moreover,enhanced media viewing capabilities such as quick start, low latency,and faster seeking capabilities can be provided to remote devices.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a communication network connected withprocessing devices.

FIG. 2 shows an example of interactions between a Web server, Web cache,and a device.

FIG. 3 shows another example of interactions between a Web server, Webcache, and a device.

FIG. 4 shows an example of a media content distribution process.

FIG. 5 shows an example of a process to request and obtain mediacontent.

FIG. 6 shows a media document example that includes message headers andcorresponding sample payloads.

FIG. 7 shows an example of a message header in a hint sample.

FIG. 8 shows an example of a server interacting with a device.

FIG. 9 shows another example of a media content distribution process.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 shows an example of a communication network connected withprocessing devices. Processing devices such as network endpoints 105,110, 120, 125, 130, 135 can connect to a communication network 115 suchas the Internet or a Local Area Network (LAN). Various examples ofendpoints include processing devices such as a mobile phone, personalcomputer 105, 110 or a computer such as a server 120, 125, 130, 135. Anendpoint can include one or more processors that can be programmed orconfigured to perform one or more operations mentioned in the presentdisclosure. In some implementations, a processor can include multipleprocessors or processor cores. A network endpoint can be identified as aclient, a server, or both, but in any case, a network endpointnecessarily includes some hardware since it includes a physical device.

Endpoints 105, 110, 120, 125, 130, 135 can access electronic documentssuch as media documents. An electronic document (which for brevity willsimply be referred to as a document) does not necessarily correspond toa file. A document may be stored in a portion of a file that holds otherdocuments, in a single file dedicated to the document in question, or inmultiple coordinated files. The document need not be a text file or adocument in the sense of a word processor. The document can includeaudio, video, images, and data content. In other examples, the documentcan be any audio, video, image, or data file. Also the document can bestreaming versions of the aforementioned document types.

In some implementations, a first server 120 is configured to handleinitial media content requests from a computer 105, 110. In someimplementations, a second server 125 is configured to stream mediacontent. For example, a computer 105, 110 can communicate with the firstserver 120 to retrieve access information regarding a media stream. Theaccess information can include contact information for the second server125. The computer 105, 110 can use the access information to communicatewith the second server 125, including requesting media fragments of themedia stream.

Server clusters 130, 135 can be configured as Web caches, e.g., HTTPbased caches configured to store Web content. A server cluster 130, 135can communicate with different endpoints connected to the network 115.In some implementations, a server cluster 130, 135 can include one ormore servers in a rack-mount configuration. In some implementations, aserver cluster can include multiple servers located in respectivedifferent physical locations with separate physical connections to thenetwork 115. In some implementations, a content distribution network(CDN) can include one or more server clusters 130, 135, which can beconfigured as a Web cache.

Endpoints 105, 110, 120, 125, 130, 135 can establish connections withother endpoints 105, 110, 120, 125. For example, servers 120, 125, 130,135 can establish connections with other servers 120, 125, 130, 135 orwith computers 105, 110. Likewise, computers 105, 110 can establishconnections with other computers 105, 110 or with servers 120, 125. Insome implementations, TCP/IP can be used to transport data betweennetwork endpoints 105, 110, 120, 125, 130, 135. In some implementations,network endpoints 105, 110, 120, 125 can communicate with each otherusing a protocol stack such as RTMP over TCP/IP. For example, a computer105 can receive a media stream from a server 120 using RTMP. In anotherexample, a computer 110 can request and receive different portions of amedia stream from a server 125, 135 using HTTP.

A server can process a Web based request from a client device thatrequests media content. In some implementations, a server clustercontaining multiple servers can process requests from client devices ina distributed fashion. A server can stream media content over a networksuch as the Internet to a client device. In some implementations,streaming media content can include accessing a segment of media contentsuch as a movie segment and sending at least a portion of the segment tothe client device in response to a HTTP request. The segment can includeone or more media data samples such as audio samples, video samples, andother data samples such as text and graphics. A server can access adocument such as one based at least in part on a MPEG4 format to obtainmedia data samples.

In some implementations, a server can fragment a MPEG4 formatteddocument, such as a movie, into multiple fragments in accordance withthe MPEG4 specification. In some implementations, a server can storemultiple versions of documents associated with a movie, with eachversion having a different bit rate.

Some document formats such as ones based on a MPEG4 format can supportinclusion of hint information to stream or render media content. Hintinformation can include hint media data and can include a hint metadatatrack. Hint media data can include one or more hint samples. A hintsample can include a network protocol header. In some implementations, ahint sample can include a network protocol header and one or more of anaudio sample, video sample, or another type of sample such as text. Insome implementations, a hint sample can include a network protocolheader and a pointer to a sample in a different media data area in lieuof containing the sample itself.

Servers can use hint information to process media content. In someimplementations, a server can use hint media data to stream mediacontent. A hint metadata track can contain pointers to locations of hintsamples in a hint media data area of a document. In someimplementations, a server can use information in a hint metadata trackand corresponding hint media data container to stream media content.

In some implementations, servers can use hint information to streammedia content to a client device using RTMP over TCP/IP. The clientdevice can process the RTMP messages and render media content. In someimplementations, servers can use hint information to generate one ormore documents containing media content that are individuallyaddressable by a HTTP request. A client device can receive a mediastream over HTTP by requesting to receive one or more documents orportions thereof that are associated with the media stream. Suchdocuments can include RTMP message headers and corresponding media datasamples arranged for playback. The client device can receive such adocument over a HTTP connection instead of a RTMP connection and canrender the media data samples based on their corresponding RTMP messageheaders. In some implementations, a client device can be configured toprocess RTMP message headers and corresponding media data samples frommultiple stacks such as RTMP over TCP/IP or HTTP over TCP/IP.

FIG. 2 shows an example of interactions between a Web server, Web cache,and a device. A device 205 such as a computer, laptop, mobile phone, caninteract with a Web server 215 to request media content. A Web server215 can provide information to the device 205. The device 205 canrequest one or more portions of the media content. In someimplementations, the device 205 can request a fragment such as a moviefragment associated with the media content. In some implementations, thedevice 205 can request a segment associated with the media content. Insome implementations, a segment can include one or more fragments. Insome implementations, the device 205 can request one or more fragmentsassociated with the media content. A fragment can include one or moreframes associated with the movie. In some implementations, the device205 can request a range of frames associated with the media content.

In some implementations, the Web server 215 can direct the device 205 toa server such as a server configured to cache content, e.g., Web cachenode 210, to retrieve at least some of the media content. In someimplementations, a networked server associated with the Domain NameSystem (DNS) of the Internet can automatically redirect the device 205to a Web cache node 210 situated in proximity to the device 205 based onnetwork topology.

FIG. 3 shows another example of interactions between a Web server, Webcache, and a device. A Web server 315 can push media content to a Webcache node 310 configured to store media content for future access. Insome implementations, a Web cache 310 can intercept a request from adevice 305 for specific media content, which can cause the Web cachenode 310 to pull information from the Web server 315 to service therequest. The Web cache node 310 can pull information from the Web server315 based on one or more factors. For example, the Web cache node 310can experience an initial cache miss where the Web cache 310 does notyet have the media content associated with the request. In someimplementations, the Web cache node 310 can forward a request to the WebServer 315 to receive content to cure a cache miss. In another example,stored information in the Web cache node 310 can require one or morerefreshes. For example, stored media content in the Web cache node 310can be time sensitive and can require periodic updates such as additionsto the content, e.g., storing media associated with a live event. Insome implementations, a Web server 315 can send media content tomultiple Web cache nodes 310, 312. A device 305 can receive differentportions of a media stream from one or more Web cache nodes 310, 312.

FIG. 4 shows an example of a media content distribution process. Adistribution process can include storing media content associated with amedia stream (405). The server cluster can include one or more serversin one or more physical locations. In some implementations, a server canupload media content to a server cluster that includes one or moreservers such as a Web Cache node. In some implementations, a server cansend media content to a server cluster that has forwarded to the servera media content request from a remote device.

Media content can include fragments associated with a media stream. Afragment can include stream messages. A stream message can include amessage header and a corresponding media data sample, such as an audiosample, video sample, or text sample. A message header can include amessage stream identifier, a message type identifier, a timestamp, and amessage length value.

The distribution process can include receiving a request associated witha media stream from a remote device using a Web protocol (410). In someimplementations, a Web protocol can include HTTP. In someimplementations, a Web protocol can include HTTP with one or moresecurity features such as Hypertext Transfer Protocol Secure (HTTPS). Insome implementations, a server can cause a processing device such as aWeb cache to handle receiving a HTTP based request associated with amedia stream.

The distribution process can include sending access informationassociated with the media stream to the remote device using the Webprotocol (415). Access information can include fragment sequencinginformation to facilitate individual fragment retrieval by a remotedevice. In some implementations, a Web server can cause a server clusterto act on its behalf. For example, a server cluster such as one or moreWeb caches configured to store media content associated with a mediastream can receive a request associated with the Web server and can sendassociated access information to the remote device.

A server can send access information to a remote device based on arequest for content such as a movie. Access information can include oneor more network addresses of respective sources for one or moredocuments associated with the requested content and can include fragmentsequencing information such as bootstrapping information to assistfragment retrieval. In some implementations, media content can beassociated with multiple media assets, the access information caninclude respective fragment sequencing information and can include anedit list.

Bootstrapping information can include fragment run information. For eachcontinuous run of one or more fragments with the same duration, aparticular run information entry can include a value denoting the numberof fragments associated with a run and a fragment duration. Runinformation entries can be listed in a data structure in an order ofplay.

In some implementations, media content can include multiple separateruns of fragments with the same or different fragment durations. In athree run example, run information can include a first fragment durationand a count of fragments associated with the first run, a secondfragment duration and a count of fragments associated with the secondrun, and a third fragment duration and a count of fragments associatedwith the third run. In some cases, the first and third fragment runs canhave the same fragment duration, whereas the second fragment run can beof a different fragment duration.

A device can use access information such as bootstrapping information toseek. In some implementations, a technique for seeking can includeconverting between a media content time offset and a correspondingfragment index based on access information. The seeking technique caninclude requesting a fragment based on the conversion.

The distribution process can include processing a fragment request fromthe remote device that identifies one or more of the fragmentsassociated with the media stream (420). In some implementations, afragment request can include a fragment index to request a specificfragment in the media stream. In some implementations, a fragmentrequest can include a segment index and a fragment offset pairing inlieu of a fragment index. A fragment offset in such a fragment requestcan refer to a specific fragment within the segment identified by acorresponding segment index.

The distribution process can include sending the one or more identifiedfragments to the remote device using the Web protocol (425). In someimplementations, different portions of a distribution process can takeplace at one or more servers. For example, a server can send mediacontent to one or more servers which are configured to handle requestsfor specifics fragments.

In some implementations, access information can include one or more of:movie identifier, live broadcast indicator, media time information,version indicator, server network address, digital rights managementinformation, a segment run table, and a fragment run table. Segment andfragment run tables can provide information to access media content suchas a movie partitioned into multiple segments.

A fragment run table can describe fragments associated with mediacontent. In some implementations, a fragment run table entry can includea first fragment index and a fragment duration for one or more fragmentsassociated with the table entry. For example, a first fragment index canindicate an index value associated with the first fragment of acontinuous run of one or more fragments that have the same duration. Insome implementations, a fragment run table entry can include a valuedenoting the number of fragments associated with a run and a fragmentduration.

A segment run table can describe segments associated with media content.In some implementations, a segment run table entry can include a firstsegment index and a count of one or more fragments associated with thetable entry. For example, a first segment index can indicate an indexvalue associated with the first segment of a continuous run of one ormore segments with similar characteristics, e.g., a run of segmentshaving the same count of fragments. In some implementations, a segmentrun table entry can include a value denoting the number of segmentassociated with a run and a fragment count.

A device can access media content such as a movie content or a livemedia stream using one or more HTTP requests to one or more servers. Adevice can receive data over one or more HTTP connections. The receiveddata can include access information, and headers and correspondingsample payloads such as audio and video data associated with therequested media content. In some implementations, a device can receiveaccess information that includes identities of one or more Web caches.In some implementations, a device can request multiple fragments fromdifferent Web caches in a concurrent fashion. The device can assemblethe fragments to render at least a portion of the media stream.

FIG. 5 shows an example of a process to request and obtain mediacontent. A device can request and receive access information associatedwith specific media content such as a movie or a live media stream(505). Access information can include contact information for a serverstoring the media content associated with the request. Accessinformation can include a fragment run table. In some implementations,access information can include a fragment run table and a segment runtable.

The device can obtain a time index associated with the media content(510). In some implementations, the device can display a media playbackcontrol user interface region The region can include a movablepositioning bar to display a playback status. A user can move thepositioning bar to select an earlier or future time index for mediaplayback. In some implementations, the device can determine a time indexbased on a position of the positioning bar relative to the userinterface region.

The device can determine a fragment index based on the accessinformation and the time index (515). Determining a fragment index caninclude computing a fragment index based on a fragment run table and thetime index.

In some implementations, media content can be stored in differentsegments, e.g., a segment document that contain fragments. The devicecan convert a fragment index into a segment index and fragment offsetpairing to access a specific segment document and fragment therein. Thedevice can determine a segment index and fragment offset based on theaccess information and the fragment index (520). Determining a segmentindex can include computing a segment index based on a segment run tableand the fragment index. Determining a fragment offset can includecomputing an offset of a fragment in a segment corresponding to thesegment index, with the fragment corresponding to the fragment index.

The device can request a specific fragment by the segment index andfragment offset (525). In some implementations, the device can request aspecific fragment identified by a fragment index. The device can displayone or more frames associated with the time index and future timeindices (530). For example, the device can start playback of a mediastream at the time index. In some implementations, media playback caninclude determining additional fragments to request. In someimplementations, media playback can include determining additionalsegments to request.

Some requests can include a web address that contains a resourceidentifier. For example, a request can include a Uniform ResourceIdentifier (URI). In some implementations, a URI can identify a documentthat contains a segment. In some implementations, a URI based requestcan include a Uniform Resource Locator (URL) to identify a fragment of aspecific segment. For example, an endpoint can use a URL such as“http://<server>/<content_identifier>/seg27#frag4” to request the 4thfragment of the 27th segment associated with the media contentidentified by <content_identifier>. In another example, an endpoint canuse a URL such as “http://<server>/<content_identifier>/s11-f5” torequest the 5th fragment of the 11th segment associated with the mediacontent identified by <content_identifier>. In another example, anendpoint can use a URL such as “http://<server>/seg23?fragment=5” torequest the 5th fragment of the 23rd segment where the server isconfigured to associate incoming requests with a media content streamthat has been pre-arranged such as a live media stream. In yet anotherexample, an endpoint can use a URL such as “http://<server>/fragment270”to request a fragment index of 270.

A server can convert a document without hint information to a documentthat includes hint information. A server can use a format such as onedescribed herein to generate a document with hint information. In someimplementations, a server can replace media data in a document with hintinformation. For example, a server can use a non duplication mode tostrip out media data such as video media data and audio media data andreplace them with hint media data that contains their respective mediasample data. In some cases, the server can modify one or more additionalportions of the original document to reflect this change. In some othercases, the server can remove the metadata tracks to generate a documentcontaining solely message headers and corresponding sample payloads. Insome implementations, a standalone software routine can add hintinformation to a document, which can be placed on one or more serversfor distribution.

In some implementations, a hint sample can include a RTMP messageheader. A corresponding RTMP hint metadata track can include a pointerto a hint sample, containing a RTMP message header, in RTMP hint mediadata. Hint media data can include hint samples associated with differentmedia data sample types. In some implementations, a document can includea container of multiplexed hint media data including RTMP messageheaders and corresponding payload information. Payload information caninclude a payload prepared for transmission or a pointer to obtain datato construct a payload portion of a message. For example, hint mediadata in a document can include multiplexed audio and video informationassociated with a movie. RTMP packet information in a hint media datacontainer can be arranged by timestamp.

FIG. 6 shows a media document example that includes message headers andcorresponding sample payloads. A document can include a media segment,which can include hint media data 615. Hint media data 615 can includemultiple hint samples 620, 625. A first hint sample 620 can include aheader such as a RTMP message header 630 and a corresponding payloadincluding an audio sample 632. A second hint sample 625 can include amessage header such as a RTMP message header 635 and a correspondingpayload including a video sample 637. In some implementations, adocument can include one or more containers of media data that includeduplicative samples corresponding to samples in the hint media data 615.

In some implementations, a document can include a movie box 640 todescribe the contents of hint media data 615. The movie box 640 caninclude metadata tracks such as an audio track 645, a video track 650,and a RTMP hint track 655. Movie box 640 is not limited to describingmovie data, but can describe other content. Metadata tracks 645, 650,655 can correspond to one or more media types such as video, audio,text, or hint. In some implementations, various metadata tracks 645,650, 655 can include pointers to locations of samples of media data inthe document. In some implementations, a movie box 640 can includeinformation about random access samples in one or more media datacontainer.

The RTMP hint track 655 can include pointers to locations of associatedhint samples 620, 625 in hint media data 615. In some implementations,hint media data 615 can interleave hint samples 620, 625 associated withdifferent media types. For example, a hint sample associated with videocan be followed by a hint sample associated with audio.

The audio track 645 can include one or more pointers 647 to respectivesample locations within a media data container. For example, the audiotrack 645 can include a pointer 647 to a location of an audio sample 632situated in a hint sample 620. The video track 650 can include one ormore pointers 652 to respective sample locations within a media datacontainer. For example, the video track 650 can include a pointer 652 toa location of a video sample 637 situated in a hint sample 625.

In some implementations, a document can include pointers synchronizedfor playback. In some implementations, pointers can be synchronized toan event or a specific time duration in media playback or streaming. Forexample, pointers 647, 652 in the audio and video tracks 645, 650 can besynchronized to an event. Multiple hint pointers 657, 659 in the hinttrack 655 can be synchronized to the same event based on theircorresponding samples 632, 637 being related to the event.

FIG. 7 shows an example of a message header in a hint sample. A hintsample can include a message header such as a RTMP message header and acorresponding payload. A RTMP message header can include a messagestream identifier 710, a message type identifier 715, a timestamp 720,and a message length value 725. In some implementations, a correspondingpayload including data can follow a RTMP message header in a hintsample.

FIG. 8 shows an example of a server interacting with a device. A server800 can send data via a connection 808 to a networked endpoint 802 suchas a laptop or a mobile device. The server 800 and endpoint 802 can useHTTP over the connection 808 to transact requests and media content. Insome implementations, the endpoint 802 can use multiple connections withone or more servers to access and receive media information.

The server 800 can access a document 804, that includes hint media data,via a data interface 806 such as a memory bus, network interface, or adisk drive interface. Hint media data can include multiple hint samples810, 812, 814, 816 with various types of sample payloads, e.g., video,audio, text. In some implementations, hint samples 810, 812, 814, 816are properly formatted RTMP messages. In some implementations, theserver 800 can access the document 804 stored on a disk drive and canstore the document 804 in memory such as a random access memory forfuture access. Hint samples 810, 812, 814, 816 in the document 804 canbe arranged in an order sequence appropriate for playback, such as atimestamp order. Further, hint samples 810, 812, 814, 816 in thedocument 804 can be partitioned 820 into multiple fragments. In someimplementations, different fragments can be stored separately.

Hint samples with different sample payload types can be multiplexed intoa single container of hint media data, which can increase serverthroughput. In some implementations, a single hint segment caninterleave different media types. For example, a hint data container caninclude hint sample payloads that respectively alternate between two ormore media types. In some cases, a hint data container can includemultiple video payloads followed by an audio payload.

The endpoint 802 can send requests 830, 835 for specific fragments ofone or more media content segments. In some implementations, an endpoint802 can request an entire media content segment. The server 800 can sendfragments 840, 845 to the endpoint 802 based on the endpoint's requests.Fragments 840, 845 can include one or more of hint samples 810, 812,814, 816. The endpoint 802 can render media content in an order based onRTMP timestamps in respective RTMP messages, e.g., hint samples 810,812, 814, 816, received in one or more fragments 840, 845. In someimplementations, the server 800 can perform one or more byte swapoperations to prepare data packets containing fragments 840, 845 fortransmission over a network. The endpoint 802 can receive fragments 840,845 and can render media content based on the received fragments 840,845.

FIG. 9 shows another example of a media content distribution process. Adistribution process can include transacting access informationassociated with a media stream (905). Access information can includefragment sequencing information to facilitate individual retrieval offragments associated with the media stream using a uniform resourceidentifier via a processing device configured to cache content. Thedistribution process can include transacting one or more of thefragments associated with the media stream to facilitate a delivery ofmedia content (910). In some implementations, transacting informationcan include sending data to one or more processing devices such as aserver. In some implementations, transacting information can includereceiving data from one or more processing devices. In someimplementations, transacting information can include receiving data andsending data.

A network endpoint can run one or more applications that include supportfor RTMP such as ADOBE® FLASH® Player and/or ADOBE® AIR® software,available from Adobe Systems Incorporated, of San Jose, Calif. Forexample, some servers can use RTMP to stream ADOBE® FLASH® content. Insome implementations, a server such as an one configured as an ADOBE®FLASH® Media Interactive Server (FMS) can stream media content to anendpoint running ADOBE® FLASH® Player. A FMS can access a mediainformation such as a FLASH® Video (e.g., F4V) document to obtain mediacontent. A F4V based document can include media content and can be inaccordance with an International Organization for Standardization (ISO)Base Media File Format. In some implementations, a FMS can use RTMP tostream media content to a ADOBE® FLASH® Player. In some implementations,a server such as a FMS configured to use HTTP can stream media contentto a player using HTTP over TCP/IP.

In some implementations, a server can use MPEG4 movie fragments, RTMPbased hint information, and access information such as a bootstrappingto provide streaming media access and connect to a client device using aWeb protocol such as HTTP. A HTTP media stream can include a stream offragments and can multiplex data such as audio and video. A fragment caninclude ADOBE® FLASH® content and can include media data samples ofdifferent media types and message headers multiplexed in time order inone or more RTMP streams. In some implementations, a server can insertadvertisements into a media stream. Web caches such as a HTTP based Webcache can store HTTP communications from a server for faster access.

A server can receive additional data associated with a media stream,such as receiving additional frames in a live video broadcast. Forexample, the server can produce one or more additional fragments basedon an incremental media data addition to the media content. The servercan provide updated access information to a remote device to reflect theone or more additional fragments. In some implementations, the servercan upload the additional fragments to a HTTP based Web cache.

In some implementations, a server can access protected media content ina document. In some implementations, a server can protect media contentbefore distribution to clients. In some implementations, a server canuse one or more digital rights management (DRM) techniques to controlaccess privileges associated with media content. In someimplementations, a server can encrypt media content and controldistribution of associated key material to decrypt said media content.In some implementations, a server can use an ADOBE® DRM system,available from Adobe Systems Incorporated of San Jose, Calif.

In some implementations, a hint sample can include multiple media datasamples. In some implementations, multiple hint samples can include thesame media data sample. In some implementations, a container of hintmedia data can include hint samples with pointers to media data samplesand can include hint samples with embedded media data samples. In someimplementations, hint samples can be transmitted to a client, which canuse the hint samples to render media content. In some implementations, aserver can transmit hint samples to an intermediate endpoint which canuse the hint samples to deliver media content to a client. Anintermediate endpoint can use different communication techniques such asdifferent network protocols for sending and receiving hint information.In some implementations, a server can add hint samples to a document andcan cache hint samples for future use. In some implementations, segmentand fragment run tables can be used with a multiplexed media format suchas MPEG-2 Transport Stream (MPEG-2 TS). In some implementations, segmentand fragment run tables can be used with a media format that providesfor fragments with mixed samples and for fragments with or without hintinformation.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A method comprising: forming a request to accessa media stream at a client device; using access information, by theclient device, that comprises fragment sequencing information tofacilitate individual retrieval of fragments associated with the mediastream using a uniform resource identifier via a respective cache ofmedia of the media stream, the fragments comprising stream messages inwhich separate ones of the stream messages include a media data sampleand a message header that includes a message stream identifier, atimestamp, and a message length value; and consuming the media datasamples of the retrieved fragments by the client device.
 2. A method asdescribed in claim 1, wherein the message header further includes amessage type identifier.
 3. A method as described in claim 1, furthercomprising receiving the access information via a network responsive tocommunicating the request to access the media stream.
 4. A method asdescribed in claim 1, further comprising receiving the one or morefragments using a Hypertext Transfer Protocol (HTTP).
 5. A method asdescribed in claim 1, wherein the access information comprisesidentities of a plurality of devices that are configured to serve one ormore fragments associated with the media stream and the accessinformation is configured to cause a remote device to request differentfragments from different devices for assembly to play at least a portionof the media stream.
 6. A method as described in claim 1, furthercomprising receiving updated access information that reflects one ormore additional fragments that were generated based on an incrementalmedia data addition to the media stream.
 7. A method as described inclaim 1, wherein the fragment sequencing information comprises one ormore fragment time durations and a number of fragments associated witheach fragment time duration, the fragment sequencing information beingarranged to indicate a fragment play order.
 8. A method as described inclaim 1, wherein the message header is formatted in accordance with aReal-Time Messaging Protocol (RTMP).
 9. A method as described in claim1, wherein one or more of the fragments comprise a Real-Time MessagingProtocol (RTMP) message associate with audio data interleaved with aRTMP message associated with video data.
 10. A method as described inclaim 1, wherein the access information comprises a segment run tableconfigured to identify runs of segments and a fragment run tableconfigured to identify runs of fragments, the fragment run tablesincluding the fragment sequencing information.
 11. A system implementedby at least one device and configured to perform operations comprising:receiving one or more fragments associated with a media stream, thefragments comprising stream messages in which separate ones of thestream messages include a media data sample and a message header thatincludes a message stream identifier, a timestamp, and a message lengthvalue; and storing the one or more fragments in a cache that isaccessible via a uniform resource identifier using access informationassociated with the media stream, the access information comprisingfragment sequencing information configured to facilitate individualretrieval of fragments associated with the media stream using theuniform resource identifier.
 12. A system as described in claim 11,wherein the message header further includes a message type identifier.13. A system as described in claim 11, further comprising sending theone or more fragments for receipt by a client device via a networkresponsive to a request received from the client device, the sendingfacilitating access of the client device to the media stream.
 14. Asystem as described in claim 11, wherein the fragment sequencinginformation comprises one or more fragment time durations and a numberof fragments associated with each fragment time duration, the fragmentsequencing information being arranged to indicate a fragment play order.15. A system as described in claim 11, wherein the access informationcomprises a segment run table configured to identify runs of segments ora fragment run table configured to identify runs of fragments, thefragment run tables including the fragment sequencing information.
 16. Asystem implemented by at least one device and configured to performoperations comprising: fragmenting media data to form one or morefragments for communication as a media stream, the fragments comprisingstream messages in which separate ones of the stream messages include amedia data sample and a message header that includes a message streamidentifier, a timestamp, and a message length value; and generatingaccess information that comprises fragment sequencing informationconfigured to facilitate individual retrieval of the fragmentsassociated with the media stream using a uniform resource identifier viaa respective cache of media of the media stream, the fragmentscomprising stream messages in which separate ones of the stream messagesinclude a media data sample and a message header that includes a messagestream identifier, a timestamp, and a message length value.
 17. A systemas described in claim 16, wherein the message header further includes amessage type identifier.
 18. A system as described in claim 16, furthercomprising generating updated access information to reflect one or moreadditional fragments that were generated based on an incremental mediadata addition to the media stream.
 19. A system as described in claim16, wherein the access information comprises a segment run tableconfigured to identify runs of segments and a fragment run tableconfigured to identify runs of fragments, the fragment run tablesincluding the fragment sequencing information.
 20. A system as describedin claim 16, wherein the access information comprises identities of aplurality of devices that are configured to serve one or more fragmentsassociated with the media stream and the access information isconfigured to cause a remote device to request different fragments fromdifferent devices for assembly to play at least a portion of the mediastream.